[00:00.000 --> 00:06.080]  This is Jeremy Gosney. I'm coming to you live from Password Village HQ2 in sunny Austin, Texas.
[00:06.080 --> 00:08.400]  Today's high is 104 degrees.
[00:08.880 --> 00:15.200]  So today I'm going to be talking to you about what it's like cracking at Terrahash's scale.
[00:16.720 --> 00:18.920]  I'm not going to make this a vendor pitch in any way.
[00:18.920 --> 00:25.900]  I'm going to focus on the technical challenges that we encounter as we support clients
[00:25.900 --> 00:28.920]  who have hundreds of GPUs in their clusters.
[00:30.620 --> 00:35.080]  When I first started on this, I thought 100 GPUs was a really big deal.
[00:35.640 --> 00:39.660]  Now we tend to view 100 GPUs as like, meh.
[00:40.660 --> 00:43.300]  It's good, but it's kind of middle of the road.
[00:47.260 --> 00:51.880]  There's a lot of distributed solutions for HashCat out there.
[00:52.360 --> 00:58.440]  But as far as I know, Terrahash is the only one that aims to operate at warehouse scale.
[00:58.440 --> 01:05.340]  What I mean by that is, more so with the most recent version, but with every version that passes,
[01:05.340 --> 01:10.020]  we aim to embrace the concepts of data center as a computer.
[01:10.560 --> 01:16.820]  While we currently don't have any clients who have an entire warehouse full of password cracking hardware,
[01:16.820 --> 01:19.300]  that's the point where we're trying to go.
[01:20.880 --> 01:27.720]  We try to enable the software to be at that level, even if our clients are not quite at that level.
[01:30.560 --> 01:38.400]  I'm going to go through the history of how I got started in massive distributed cracking.
[01:38.520 --> 01:46.540]  And I'm going to go through the technical challenges that we faced as we continued to evolve our hashtag distributed solution.
[01:47.580 --> 01:49.600]  I'm going to start the slide deck.
[01:57.330 --> 01:58.810]  All right.
[02:01.480 --> 02:07.300]  So, in the beginning, I kind of got into this because I was messing around with distributed computing.
[02:07.300 --> 02:16.720]  I was doing stuff with MPI, some other MOSIX software, rocks clusters, things along those natures.
[02:17.820 --> 02:19.840]  But all that was all CPU-based.
[02:19.840 --> 02:29.900]  And then when GPU cracking came about, we started looking for solutions that would enable us to distribute GPU workloads.
[02:29.940 --> 02:34.840]  So, I found a piece of software at a Hebrew university called VirtualCL.
[02:35.160 --> 02:44.060]  And I did a presentation on a 25 GPU proof of concept cluster that me and Bitweasel put together at passwords12 in Oslo.
[02:44.300 --> 02:49.740]  And when I woke up the next morning after giving that presentation, we had gone viral.
[02:49.840 --> 02:56.060]  On the front page of Slashdot, Gizmodo, Boing Boing, NBC News, The Register, we were everywhere.
[02:57.140 --> 03:01.480]  And in hindsight, I really should have used a lot more than 25 GPUs.
[03:01.480 --> 03:05.300]  I mean, we had more GPUs, that's just what we chose to dedicate to this proof of concept.
[03:05.300 --> 03:12.540]  And had I known in advance in any way that the general public would react how they did to this proof of concept,
[03:12.540 --> 03:15.160]  I would have used like three times the number of GPUs.
[03:17.060 --> 03:22.940]  So just, you know, some of the headlines that came out the next morning that I woke up to,
[03:22.940 --> 03:25.980]  which was pretty surreal because I didn't really see this as a big deal.
[03:27.360 --> 03:35.500]  So, VirtualCL was cool at the time because it was the first solution that I knew of that enabled us to
[03:36.300 --> 03:42.820]  sort of transparently distribute OpenCL jobs across the arbitrary cluster.
[03:42.820 --> 03:47.080]  So what VirtualCL does is it provides a virtual OpenCL platform.
[03:48.620 --> 03:54.080]  And the target software, in theory, doesn't need to be aware it's communicating with a virtual platform.
[03:54.080 --> 04:00.400]  It just sees any, you know, it sees VirtualCL as any other, you know, OpenCL platform,
[04:00.400 --> 04:03.160]  just like AMD, NVIDIA, Intel, what have you.
[04:04.560 --> 04:10.200]  So the target software just simply links against libOpenCL, which is provided by VCL,
[04:10.200 --> 04:15.140]  which then communicates to a broker daemon, and then that broker daemon distributes to the agent daemons,
[04:15.140 --> 04:17.400]  and the agent daemons, which are installed on the compute nodes,
[04:18.100 --> 04:24.440]  actually communicate with the real OpenCL library, be it AMD, NVIDIA, whatever.
[04:26.040 --> 04:29.680]  So in theory, all this is transparent to the end application.
[04:29.680 --> 04:32.420]  But in practice, it didn't quite work that way.
[04:33.380 --> 04:38.440]  We had to have a special fork of OCLHashCat called VCLHashCat,
[04:38.440 --> 04:42.160]  which had some workarounds for VCL quirks and such.
[04:43.280 --> 04:47.760]  It also required InfiniBand, because it was really latency sensitive.
[04:47.980 --> 04:53.140]  It ended up being a very chatty protocol, and it required essentially real-time communication.
[04:53.400 --> 04:55.000]  Bandwidth wasn't so much of an issue.
[04:55.000 --> 04:59.780]  It pretty much consistently pulled, you know, less than one gig of bandwidth,
[04:59.780 --> 05:03.600]  but it was really sensitive to latency.
[05:03.960 --> 05:08.040]  Even just a couple milliseconds was acceptable,
[05:08.040 --> 05:10.760]  but as you start getting up into double digits of latency,
[05:10.760 --> 05:13.620]  you had a noticeable impact on hash rate.
[05:14.480 --> 05:16.920]  And of course, InfiniBand hardware is expensive.
[05:16.920 --> 05:19.300]  It's way more expensive than Ethernet.
[05:20.240 --> 05:23.520]  VCL was also closed source, and it required frequent updates.
[05:23.520 --> 05:28.120]  So this is back in the time, this is 2012 through 2013,
[05:28.120 --> 05:33.300]  where every GPU driver that was released broke some shit.
[05:33.300 --> 05:36.480]  And then we had to go back in and implement workarounds and fixes,
[05:36.480 --> 05:40.540]  whatever the fuck AMD or NVIDIA broke in their driver that day.
[05:40.540 --> 05:42.660]  And VCL was no exception.
[05:42.660 --> 05:48.600]  So with VCLHashCat, we had to have our own workarounds for VirtualCL,
[05:48.600 --> 05:51.780]  and then the VCL team, or VirtualCL team,
[05:51.780 --> 05:56.560]  had to turn around and implement any workarounds for any new driver releases
[05:56.560 --> 05:58.440]  every time they were released.
[05:58.440 --> 06:02.280]  Which is a pretty substantial job.
[06:02.560 --> 06:05.180]  The problem is, VirtualCL was created by grad school students.
[06:05.180 --> 06:08.850]  This was a grad school project, and they created this their senior year.
[06:09.360 --> 06:13.680]  And when they graduated in spring 2013, that was it.
[06:13.680 --> 06:15.300]  The project died.
[06:15.860 --> 06:20.720]  So when the next version of FGLRX was released,
[06:20.720 --> 06:24.560]  I think that was version 13.9 in September of 2013,
[06:25.600 --> 06:30.280]  we go to install a VirtualCL cluster for our client.
[06:30.280 --> 06:32.800]  This is actually a 64 GPU cluster.
[06:33.060 --> 06:35.740]  We were just starting out, so this was actually a pretty big deal for us.
[06:35.740 --> 06:38.480]  64 GPUs was pretty fucking cool at the time.
[06:39.780 --> 06:44.480]  But we get the OS installed, we get the hardware built, all that shit,
[06:44.480 --> 06:48.980]  and then we go to install VirtualCL and FGLRX,
[06:49.500 --> 06:51.440]  and VirtualCL just shits the bed.
[06:51.440 --> 06:54.800]  So I panic and email Hebrew University,
[06:54.800 --> 06:56.760]  and I'm like, you guys really need to update VirtualCL
[06:58.240 --> 07:02.340]  for Catalyst and FGLRX 13.9.
[07:02.780 --> 07:06.680]  And they're like, well, we can't.
[07:06.680 --> 07:09.200]  Everyone who was working on this project has gone.
[07:10.660 --> 07:14.660]  And the intellectual property remains with Hebrew University and MOSIX,
[07:14.660 --> 07:18.520]  so they can just have those people work on it outside of that.
[07:18.940 --> 07:21.380]  I offered to take over the project for them,
[07:22.420 --> 07:25.040]  because it was kind of essential to what we were doing,
[07:25.040 --> 07:26.420]  and I didn't want to see the project die.
[07:26.420 --> 07:27.680]  I thought it was really neat.
[07:28.940 --> 07:32.580]  But the professor over there who was in charge of the project
[07:32.580 --> 07:35.180]  was not amenable to that idea.
[07:35.180 --> 07:37.000]  He actually had pretty grave concerns
[07:37.000 --> 07:39.460]  when we first started using VirtualCL for password cracking,
[07:39.460 --> 07:43.140]  because he envisioned us turning the world into a giant botnet.
[07:43.540 --> 07:45.860]  And of course, that wasn't really on our agenda,
[07:45.860 --> 07:47.500]  at least not at that time.
[07:47.500 --> 07:52.340]  But yeah, he really didn't like the idea
[07:52.340 --> 07:57.660]  of his VirtualCL shit ending up in our hands.
[07:58.280 --> 08:02.000]  So we were left high and dry without a software solution,
[08:02.000 --> 08:04.640]  and so we desperately needed something.
[08:04.700 --> 08:07.500]  So we conceived the idea of Hashtag.
[08:09.700 --> 08:12.380]  Initially, we thought about making our own VirtualCL clone,
[08:12.380 --> 08:16.680]  since Hebrew University was not going to give us the VirtualCL source code.
[08:16.680 --> 08:18.180]  And we're like, well, fuck it.
[08:18.180 --> 08:21.600]  We'll start our own VirtualCL with hookers and blackjack.
[08:22.320 --> 08:26.380]  But the more we actually sat down and thought about what that entailed,
[08:26.380 --> 08:30.700]  we determined the level of effort for that was just way too high.
[08:30.760 --> 08:34.100]  And the timeframe in which we needed it was way too short.
[08:34.500 --> 08:37.800]  We literally had hardware in-house that we're building for clients
[08:37.800 --> 08:41.680]  who now have no software solution because of VirtualCL.
[08:45.970 --> 08:47.950]  So we decided to make a distributed wrapper.
[08:47.950 --> 08:50.470]  And that's when we came up with the idea of Hashtag.
[08:51.190 --> 08:53.570]  Like I said, we had to iterate really quickly on this.
[08:53.570 --> 08:59.190]  So we went from a whiteboard session to production in less than two months.
[08:59.230 --> 09:03.330]  And this was basically me and Tom Steele locking ourselves in the office
[09:04.850 --> 09:07.430]  and ordering over $100 for the Taco Bell
[09:07.950 --> 09:11.310]  and just banging this out as fast and furious as we could.
[09:12.630 --> 09:16.010]  It had a traditional client-server-agent architecture.
[09:16.650 --> 09:18.570]  And it was Hashtag focused,
[09:18.570 --> 09:20.690]  but it actually had a generic plugin interface.
[09:20.990 --> 09:25.230]  We had plugins for... and this was before there was just one Hashtag.
[09:25.230 --> 09:33.150]  So we had Hashtag, OCL Hashtag Plus, OCL Hashtag Lite plugins.
[09:33.150 --> 09:35.090]  And then we had John the Ripper plugin.
[09:36.110 --> 09:38.970]  And then we had just a generic interface
[09:38.970 --> 09:41.650]  to where any cracking tool could be made to work with Hashtag
[09:41.650 --> 09:46.890]  as long as it adhered to a standard format that we had defined.
[09:47.350 --> 09:50.310]  And then it also had the ability to run arbitrary commands when idle.
[09:51.010 --> 09:53.690]  This was less for our clients and actually more for me
[09:53.690 --> 09:56.270]  because we were running Hashtag in-house as well
[09:56.270 --> 09:59.410]  for our cracking-as-a-service type stuff.
[10:00.470 --> 10:04.290]  And this is back in a time when GPU mining was still profitable.
[10:04.370 --> 10:06.730]  So when we weren't working on a password cracking job,
[10:06.730 --> 10:09.750]  I wanted this thing to be generating coins in the background.
[10:09.750 --> 10:14.930]  So the arbitrary idle exec command was literally just put in place
[10:14.930 --> 10:19.490]  so that people could mine Litecoin or Namecoin or whatever
[10:19.490 --> 10:22.590]  when their cluster was idle.
[10:24.090 --> 10:26.270]  So, you know, totally different time.
[10:26.670 --> 10:28.170]  But, I mean, it actually worked.
[10:28.170 --> 10:32.530]  It did exactly what it was supposed to do and it was good.
[10:33.530 --> 10:36.270]  And the whole goddamn thing was implemented
[10:36.270 --> 10:38.570]  in this fancy new language called Node.js.
[10:38.570 --> 10:44.370]  Again, this is early 2013, so Node.js was the new hotness
[10:44.370 --> 10:46.830]  and everyone was starting to move that direction
[10:46.830 --> 10:48.610]  with microservices and everything.
[10:48.850 --> 10:52.470]  It's not on this slide, but we implemented Hashtag version 1
[10:52.470 --> 10:54.270]  entirely as microservices.
[10:55.130 --> 10:59.110]  And I shit you not, there was like 23 individual packages
[10:59.110 --> 11:03.210]  that comprised Hashtag and almost all of those were microservices.
[11:05.350 --> 11:07.890]  And we went that direction because that's kind of like
[11:08.570 --> 11:09.970]  programming zeitgeist at the time, right?
[11:09.970 --> 11:12.930]  Like everyone was pushing microservices and pushing asynchronous
[11:13.370 --> 11:17.170]  and, you know, NoSQL and all that other horseshit, right?
[11:17.170 --> 11:20.190]  And in the end, it created more headaches than not
[11:20.190 --> 11:22.810]  if we just had a monolithic fucking program.
[11:23.490 --> 11:26.970]  You know, maintaining like one server binary
[11:26.970 --> 11:31.570]  and one agent binary is way simpler than like, you know,
[11:31.570 --> 11:34.270]  maintaining 23 individual microservices and making sure
[11:34.270 --> 11:35.990]  that they're all synced and communicating properly
[11:35.990 --> 11:39.230]  and, you know, that the versions are correct, you know,
[11:39.230 --> 11:40.810]  across all 23 packages.
[11:40.950 --> 11:43.310]  And there's just a lot of unnecessary headache
[11:43.310 --> 11:46.270]  that was introduced with that, that, you know,
[11:46.270 --> 11:48.510]  we attempted to manage.
[11:48.870 --> 11:51.430]  But even then, like our clients would still kind of mess it up.
[11:51.430 --> 11:56.010]  You know, like they would upgrade like 10 of 20 packages
[11:56.010 --> 11:58.030]  and then like the other ones would just be stuck at old versions
[11:58.030 --> 12:01.770]  or, you know, one demon would die and another demon would know
[12:01.770 --> 12:03.470]  that it's dead, so then it would shit the bed.
[12:03.470 --> 12:09.470]  And anyway, so yeah, no JS, microservices,
[12:09.470 --> 12:11.270]  no SQL, MongoDB, all that shit.
[12:11.370 --> 12:13.310]  I tried to use all like the latest and greatest everything
[12:13.310 --> 12:15.950]  because at the time, that seemed like the best idea.
[12:17.010 --> 12:19.710]  And also, Hashtag was more than just workload distribution.
[12:19.710 --> 12:21.290]  So a lot of our clients at the time thought
[12:21.290 --> 12:25.450]  that Hashtag was just a GUI front-end for Hashcat.
[12:25.850 --> 12:26.990]  And we're like, no, it's not.
[12:26.990 --> 12:29.530]  It's not just a web UI for Hashcat.
[12:29.530 --> 12:33.370]  It actually does like, you know, multi-user and, you know,
[12:33.470 --> 12:35.970]  workload distribution with like a really complex
[12:35.970 --> 12:37.630]  four-dimensional queuing mechanism.
[12:37.630 --> 12:38.830]  Like you're trying to explain all this to the client
[12:38.830 --> 12:40.370]  and they're just like, you know, eyes glazed over
[12:40.370 --> 12:42.310]  and like, so it's a web UI for Hashcat.
[12:42.470 --> 12:45.110]  Like fucking yes, okay, it's a web UI for Hashcat.
[12:46.090 --> 12:47.590]  But it actually did a hell of a lot more than that.
[12:47.590 --> 12:50.290]  In fact, when you installed Hashtag on your box,
[12:50.290 --> 12:53.070]  it basically hijacked it and configured it
[12:53.070 --> 12:54.990]  exactly the way that we needed it to be configured.
[12:54.990 --> 12:56.810]  It would actually provision the entire server
[12:56.810 --> 13:00.570]  from just installing like the initial meta package.
[13:01.410 --> 13:03.350]  So it's a complete stack of packages.
[13:03.350 --> 13:05.430]  They get stack, Hashtag, that's where it comes from.
[13:05.910 --> 13:08.190]  So it handled driver installation,
[13:08.190 --> 13:10.170]  driver configuration, driver updates.
[13:10.170 --> 13:11.630]  And again, this is back at a time,
[13:11.630 --> 13:13.090]  and even nowadays still kind of a challenge,
[13:13.090 --> 13:18.990]  but this is back at a time when GPU drivers
[13:18.990 --> 13:20.130]  were a pain in the ass to install.
[13:20.130 --> 13:21.210]  And then once you got them installed,
[13:21.210 --> 13:22.570]  it was a pain in the ass to update.
[13:23.150 --> 13:24.530]  So the fact that we kind of handled this
[13:24.530 --> 13:27.230]  automagically was a pretty big value add.
[13:27.970 --> 13:29.490]  Also handled XOR configuration,
[13:30.870 --> 13:32.070]  headless servers.
[13:32.350 --> 13:33.550]  This is actually during a time
[13:33.550 --> 13:35.270]  where when you had a headless server,
[13:35.270 --> 13:36.650]  you had to have fucking dummy plugs
[13:36.650 --> 13:38.530]  connected to each of the GPUs.
[13:38.530 --> 13:40.150]  There's probably very few people watching this
[13:40.150 --> 13:41.470]  who actually remember that.
[13:42.330 --> 13:44.250]  But yeah, if you had a completely headless server,
[13:44.250 --> 13:45.410]  each one of those GPUs had to have
[13:45.510 --> 13:47.190]  a special dummy plug attached to it
[13:47.650 --> 13:49.330]  that would connect the necessary pins
[13:49.330 --> 13:50.630]  to trick it to think that there's actually
[13:50.770 --> 13:51.850]  a monitor attached.
[13:51.990 --> 13:53.610]  And we actually had to run XOR
[13:53.610 --> 13:55.710]  because even though these are headless,
[14:00.570 --> 14:01.410]  they don't have GPU clock rates
[14:01.830 --> 14:03.670]  or set like power tune profiles
[14:03.670 --> 14:04.610]  or any of that.
[14:04.610 --> 14:06.610]  So even though this is a headless server,
[14:06.610 --> 14:07.770]  we still had to have dummy plugs
[14:07.770 --> 14:09.090]  to think there's a monitor attached,
[14:09.090 --> 14:10.130]  still had to run XOR,
[14:10.130 --> 14:13.190]  and still had to configure XOR properly
[14:14.010 --> 14:16.750]  to enable us to control the fans
[14:16.750 --> 14:17.970]  and control the clocks
[14:17.970 --> 14:19.630]  and all the other shit that we needed to do.
[14:20.030 --> 14:22.110]  And then along those lines,
[14:22.110 --> 14:24.290]  Hashtag also did GPU tuning for you
[14:24.290 --> 14:26.090]  and also automatic overclocking.
[14:26.090 --> 14:33.470]  And this was kind of a double-edged sword.
[14:33.510 --> 14:36.710]  So when the R9 290X came out,
[14:36.710 --> 14:38.490]  at first it was like entirely unusable
[14:39.430 --> 14:42.190]  because of the way that AMD aggressively
[14:42.190 --> 14:45.910]  implemented power tune firmware,
[14:45.910 --> 14:48.570]  I think it was 2.0 at the time.
[14:50.470 --> 14:53.490]  But the GPUs would throttle so aggressively
[14:53.490 --> 14:56.070]  that it was like way slower than the 7970.
[14:57.130 --> 14:59.050]  And it was just essentially unusable.
[14:59.050 --> 15:02.310]  So then I wrote od6config.
[15:05.310 --> 15:06.470]  Sorry, did I say PowerTune?
[15:06.470 --> 15:08.670]  PowerTune is fucking NVIDIA. Overdrive. I'm sorry.
[15:09.450 --> 15:10.770]  Oh, no, no. PowerTune is AMD.
[15:10.810 --> 15:12.570]  PowerMizer is NVIDIA. All right, whatever.
[15:13.870 --> 15:16.890]  So od6config enabled us
[15:16.890 --> 15:19.970]  to properly configure the R9 290X.
[15:20.930 --> 15:22.250]  But it was good,
[15:22.250 --> 15:23.590]  but it was still kind of mediocre.
[15:24.190 --> 15:26.010]  But at that time, when I measured it,
[15:26.090 --> 15:27.910]  it was only drawing 300 watts,
[15:27.910 --> 15:28.710]  which is important
[15:29.450 --> 15:31.890]  because our server platforms like the Brutalis
[15:32.450 --> 15:35.130]  can only handle GPUs that drop to 300 watts.
[15:35.130 --> 15:36.970]  You draw more than that, you're going to blow a fuse
[15:36.970 --> 15:38.730]  on the PCI slots on the motherboard.
[15:40.550 --> 15:44.610]  So we had a automatic overclock profile
[15:44.610 --> 15:46.730]  for the R9 290X that we pushed out to our clients
[15:47.830 --> 15:52.410]  that made the GPU draw just about 300 watts.
[15:52.410 --> 15:56.510]  And then a little bit later down the road,
[15:56.510 --> 15:58.310]  about a year, year and a half later,
[15:58.310 --> 16:01.210]  AMD pushed out a new driver update
[16:01.210 --> 16:03.530]  that actually caused the power consumption
[16:03.530 --> 16:06.730]  to go up by a good 50 to 75 watts.
[16:06.730 --> 16:09.270]  And we did not know this.
[16:09.270 --> 16:11.030]  This kind of went undetected by us
[16:11.030 --> 16:15.450]  until we started getting a rash of clients
[16:16.030 --> 16:19.530]  who were seeing dead PCIe slots on their motherboards,
[16:19.530 --> 16:21.930]  and all of them had R9 290X-based solutions.
[16:22.910 --> 16:25.230]  And I finally busted out a kilowatt
[16:25.230 --> 16:27.190]  and figured out what the fuck was going on.
[16:28.290 --> 16:30.510]  But we actually had the ability,
[16:30.510 --> 16:35.290]  through all these mechanisms that we had in Hashtag
[16:35.290 --> 16:38.670]  at the time to enable the automatic overclocking
[16:38.670 --> 16:40.490]  and configuration and all that,
[16:40.490 --> 16:43.350]  to push out a new underclock profile
[16:43.350 --> 16:45.870]  to all of our clients to try to drop
[16:45.870 --> 16:47.450]  the power consumption down to keep it
[16:47.450 --> 16:50.170]  from blowing PCIe slots on the motherboards.
[16:51.770 --> 16:52.970]  So it's both good and bad.
[16:52.970 --> 16:55.470]  The fact that we're able to automatically overclock GPUs
[16:56.150 --> 16:59.530]  to help get the most speed out of the GPUs is awesome.
[17:00.530 --> 17:02.590]  But then in that one specific scenario,
[17:02.590 --> 17:05.310]  it was bad because we started blowing PCIe slot fuses.
[17:05.450 --> 17:06.930]  But then it was good that we had this functionality
[17:06.930 --> 17:09.330]  because then we could correct it through software,
[17:09.330 --> 17:12.090]  over the air, and automatically correct it
[17:12.090 --> 17:14.010]  on all of our client systems all at the same time.
[17:15.590 --> 17:16.670]  Which saved us a lot of money
[17:16.670 --> 17:19.110]  because we were spending hundreds of thousands of dollars
[17:19.110 --> 17:20.390]  on new motherboards.
[17:21.790 --> 17:22.430]  So yeah.
[17:22.430 --> 17:24.210]  And then, of course, Hashtag also had
[17:24.210 --> 17:25.830]  cluster resource monitoring.
[17:28.330 --> 17:30.170]  So there were some drawbacks.
[17:30.170 --> 17:31.830]  Like I said, the entire Godend thing
[17:31.830 --> 17:33.050]  was written in Node.js,
[17:33.050 --> 17:35.350]  and while Node.js is asynchronous,
[17:35.350 --> 17:38.430]  it also runs on a single CPU thread.
[17:38.670 --> 17:40.430]  That may have changed at some point,
[17:40.430 --> 17:41.930]  because we're talking like 2013,
[17:41.930 --> 17:42.910]  that's seven years ago,
[17:42.910 --> 17:45.570]  and I haven't even touched Node.js since.
[17:45.730 --> 17:47.710]  But at the time, everything ran
[17:49.110 --> 17:49.430]  on a single CPU thread,
[17:49.430 --> 17:50.010]  which was stupid,
[17:50.010 --> 17:51.110]  because our cluster controllers
[17:51.110 --> 17:54.570]  had a minimum of 12 CPU cores,
[17:54.570 --> 17:55.450]  24 threads,
[17:55.450 --> 17:58.230]  and we're sitting here stuck in one thread
[17:58.850 --> 18:00.790]  on the server daemon.
[18:01.610 --> 18:02.890]  So yeah, any long-running methods
[18:02.890 --> 18:04.570]  would just totally block
[18:04.570 --> 18:05.950]  the entire application,
[18:05.950 --> 18:09.430]  and we had massive scalability issues.
[18:10.210 --> 18:13.510]  I think the very first alpha
[18:13.510 --> 18:15.010]  that we released to clients
[18:15.010 --> 18:17.510]  could only support four nodes.
[18:17.510 --> 18:18.750]  The second one,
[18:18.750 --> 18:21.830]  we were able to get up to nine or ten nodes.
[18:22.290 --> 18:23.670]  Basically, any time a client ordered
[18:23.830 --> 18:25.230]  a cluster larger than the previous
[18:25.230 --> 18:26.950]  largest cluster we had ever built,
[18:26.950 --> 18:28.950]  we ran into some kind of scalability issue,
[18:28.950 --> 18:30.130]  identified some bottlenecks,
[18:30.130 --> 18:31.870]  and had to work to remove those bottlenecks.
[18:32.870 --> 18:35.350]  Unfortunately, 90% of the time,
[18:35.350 --> 18:36.370]  the bottleneck was the fact
[18:36.370 --> 18:38.030]  that JavaScript was too slow
[18:38.030 --> 18:40.110]  for the task that we were trying to do.
[18:40.290 --> 18:41.650]  So we had to reimplement
[18:42.830 --> 18:44.110]  those bottleneck methods
[18:44.110 --> 18:45.210]  in a faster language,
[18:45.210 --> 18:47.570]  and then just shell out to them.
[18:48.350 --> 18:49.590]  So keyspace calculation
[18:49.590 --> 18:51.070]  was reimplemented in C.
[18:51.070 --> 18:53.230]  Now, calculating keyspace for a mask attack
[18:53.230 --> 18:54.550]  is really lightweight,
[18:54.550 --> 18:56.830]  but when you talk about massive dictionaries
[18:56.830 --> 18:59.090]  or rule files,
[18:59.090 --> 19:00.670]  trying to count the number of lines
[19:00.670 --> 19:02.330]  in a file with JavaScript
[19:02.330 --> 19:04.510]  is painfully slow.
[19:05.350 --> 19:06.710]  So that's largely why
[19:06.710 --> 19:09.270]  that entire portion was reimplemented in C,
[19:09.270 --> 19:11.190]  and we just shelled out to that binary
[19:11.810 --> 19:13.170]  to eliminate that bottleneck.
[19:14.310 --> 19:16.430]  We didn't dramatically reduce it.
[19:17.850 --> 19:18.590]  Regex.
[19:18.590 --> 19:20.110]  Anything Regex,
[19:20.110 --> 19:21.650]  we actually implemented,
[19:21.650 --> 19:22.850]  we had reimplemented it in Perl,
[19:22.850 --> 19:25.250]  because Perl is amazing at Regex.
[19:25.250 --> 19:28.570]  JavaScript is not very good at Regex.
[19:28.950 --> 19:30.610]  The primary thing
[19:30.610 --> 19:31.810]  that we were doing with Regex here
[19:31.810 --> 19:34.990]  was validating the hash lists.
[19:34.990 --> 19:37.510]  So a client uploads a hash list,
[19:37.510 --> 19:39.250]  and some of our clients
[19:39.250 --> 19:40.210]  wouldn't even bother throwing
[19:40.210 --> 19:42.010]  200 million hashes at hashtag
[19:42.010 --> 19:43.570]  in one job.
[19:43.970 --> 19:45.550]  But then hashtag has to sit there
[19:45.550 --> 19:50.610]  and receive the massively large hash file,
[19:50.610 --> 19:53.330]  and then validate each individual hash,
[19:53.330 --> 19:55.710]  and then stuff them in a database.
[19:56.130 --> 19:58.270]  That was a massive bottleneck.
[19:58.270 --> 19:59.010]  So we ended up reimplementing
[19:59.590 --> 20:00.730]  all that in Perl
[20:00.730 --> 20:02.830]  to try to get that down.
[20:02.830 --> 20:04.310]  And then the agent,
[20:04.310 --> 20:05.710]  the entire fucking agent,
[20:05.710 --> 20:07.410]  we ended up just implementing
[20:07.410 --> 20:08.730]  the whole thing in Perl.
[20:12.710 --> 20:13.930]  And the only reason
[20:13.930 --> 20:14.790]  we picked Perl at the time
[20:14.790 --> 20:15.930]  is because that's the only language
[20:15.930 --> 20:17.250]  we could iterate fast enough in
[20:17.250 --> 20:19.090]  to rewrite the entire damn thing.
[20:21.770 --> 20:23.090]  But yeah.
[20:23.430 --> 20:24.910]  Like I said, we also used MongoDB
[20:26.090 --> 20:29.750]  because NoSQL was the rage at the time.
[20:30.070 --> 20:32.590]  And out of all the NoSQL databases we tested,
[20:32.590 --> 20:34.450]  Mongo was the most performant.
[20:35.010 --> 20:36.790]  But then we had clients do things
[20:36.790 --> 20:39.190]  like throw 200 million hashes at hashtag.
[20:39.950 --> 20:41.490]  And that would cause
[20:42.970 --> 20:44.330]  some not only bottlenecks
[20:45.290 --> 20:47.110]  and blocking processes,
[20:47.110 --> 20:51.390]  but some really UI weirdness as well.
[20:51.530 --> 20:53.270]  They would go through the process
[20:53.270 --> 20:54.490]  of creating a job,
[20:54.490 --> 20:55.850]  uploading the hash list,
[20:55.850 --> 20:56.890]  and they would just sit there
[20:56.890 --> 20:57.790]  with the little spinner
[20:58.690 --> 21:01.350]  for a while until it timed out.
[21:01.830 --> 21:03.510]  So they'd go to create the job again
[21:03.510 --> 21:04.650]  because it timed out.
[21:04.650 --> 21:06.510]  And then they'd go back 10 minutes later
[21:09.190 --> 21:10.590]  and it was actually active and running.
[21:10.590 --> 21:12.430]  But because they went through the process four times
[21:12.430 --> 21:13.390]  trying to get it to start,
[21:13.390 --> 21:14.910]  now they have like four of the same process
[21:14.910 --> 21:16.090]  running on the cluster.
[21:16.470 --> 21:19.190]  So we threw Redis in front of Mongo
[21:19.190 --> 21:21.930]  as an in-memory caching layer.
[21:22.290 --> 21:24.330]  And that actually had a significant impact,
[21:24.330 --> 21:25.670]  like positive impact.
[21:25.850 --> 21:27.910]  And it enabled us to ingest hashes
[21:28.110 --> 21:29.230]  a lot faster.
[21:30.930 --> 21:34.570]  But still not a very great solution overall.
[21:36.190 --> 21:38.510]  The Web UI client was implemented
[21:38.510 --> 21:41.310]  as a single-page application in Angular.
[21:41.750 --> 21:43.470]  It was really clunky.
[21:43.470 --> 21:44.670]  It was really heavy.
[21:44.670 --> 21:46.090]  It was really slow.
[21:46.110 --> 21:48.350]  It constantly caused the browser to freeze
[21:48.350 --> 21:50.830]  if you left it up for more than 30 minutes.
[21:50.930 --> 21:52.910]  And it just had a ton of bugs.
[21:53.390 --> 21:54.790]  And I think probably the most annoying thing
[21:54.790 --> 21:56.370]  is if you're like a hashcat power user
[21:56.370 --> 21:57.770]  and you sit down and try to use
[21:57.770 --> 21:59.670]  the hashtag Web UI,
[21:59.670 --> 22:02.610]  it was just cumbersome as fuck.
[22:03.010 --> 22:04.550]  There's no way to move quickly
[22:04.570 --> 22:06.390]  in that UI.
[22:06.550 --> 22:07.810]  And it gets very repetitive
[22:07.810 --> 22:09.290]  and very tedious.
[22:10.370 --> 22:12.390]  So there was a lot of drawbacks
[22:12.390 --> 22:14.950]  to having that Angular Web UI.
[22:15.250 --> 22:16.770]  I absolutely hated it.
[22:16.770 --> 22:20.370]  And anyone who was really experienced
[22:20.370 --> 22:22.310]  with cracking and hashcat and stuff,
[22:22.310 --> 22:23.550]  they also hated it
[22:23.550 --> 22:26.910]  and begged us not to have that again.
[22:27.570 --> 22:29.270]  And we were just severely limited
[22:29.270 --> 22:30.830]  by what we could do in a browser.
[22:31.410 --> 22:34.130]  For non-hash formats at the time,
[22:34.130 --> 22:36.390]  we were scraping everything server-side.
[22:36.390 --> 22:38.170]  So you would actually upload a PDF
[22:38.170 --> 22:39.610]  or upload a .docx
[22:39.610 --> 22:41.870]  or whatever,
[22:41.870 --> 22:45.590]  and we would scrape the file server-side
[22:45.590 --> 22:49.190]  to get the necessary bits,
[22:49.190 --> 22:51.630]  like Office to hashcat
[22:51.630 --> 22:52.870]  and PDF to hashcat,
[22:52.870 --> 22:53.910]  shit like that.
[22:54.530 --> 22:56.190]  The problem then became,
[22:56.190 --> 22:58.310]  what do we do with things like
[22:58.310 --> 22:59.510]  TrueCrypt volumes
[22:59.510 --> 23:02.990]  or really large 7-zip volumes
[23:02.990 --> 23:04.690]  or RAR files?
[23:04.690 --> 23:06.110]  Are you really going to upload
[23:06.750 --> 23:09.410]  a 100-gigabyte disk image
[23:09.410 --> 23:10.370]  through the browser
[23:10.370 --> 23:12.170]  just so we can scrape off
[23:12.850 --> 23:14.390]  two kilobytes?
[23:14.390 --> 23:17.010]  No, that doesn't make any sense.
[23:19.690 --> 23:21.710]  So yeah, the browser was an issue.
[23:21.710 --> 23:24.450]  And then we implemented access controls,
[23:24.450 --> 23:25.970]  but there was nothing fine-grained.
[23:25.970 --> 23:28.770]  We assumed that because everyone
[23:28.770 --> 23:30.790]  operating the hashcat cluster
[23:30.790 --> 23:32.210]  was on the same team,
[23:32.990 --> 23:35.710]  that they all had equal rights
[23:35.710 --> 23:37.190]  and equal access to the cluster,
[23:37.190 --> 23:38.770]  and therefore all users are admins.
[23:38.770 --> 23:42.010]  We didn't have different roles or anything.
[23:42.010 --> 23:44.350]  It was just, if you have an account,
[23:44.350 --> 23:46.650]  then you obviously have a right to be there,
[23:46.650 --> 23:48.410]  so therefore you are an admin.
[23:50.970 --> 23:52.590]  And I think some of our clients
[23:52.590 --> 23:53.390]  didn't have a problem with that,
[23:53.390 --> 23:54.970]  because everyone on this team
[23:54.970 --> 23:57.290]  does have equal rights and access
[23:57.290 --> 23:58.510]  and privilege to the cluster,
[23:58.510 --> 23:59.730]  so this is totally fine.
[23:59.730 --> 24:01.110]  Other clients were like,
[24:01.110 --> 24:02.390]  this is absolutely unacceptable.
[24:04.810 --> 24:06.210]  And I eventually got to the point
[24:06.210 --> 24:09.010]  where the old version one code
[24:09.010 --> 24:10.970]  was completely unmanageable.
[24:11.370 --> 24:12.550]  Even just implementing
[24:12.550 --> 24:14.350]  what should have been a minor bug fix
[24:14.350 --> 24:18.210]  ended up requiring major refactoring.
[24:19.890 --> 24:20.990]  It just wasn't,
[24:20.990 --> 24:21.850]  because we threw it together
[24:21.850 --> 24:22.870]  in less than two months,
[24:22.870 --> 24:25.450]  so we didn't think of everything in advance
[24:25.450 --> 24:27.130]  that would come up,
[24:29.730 --> 24:30.770]  and try to implement new features,
[24:30.770 --> 24:32.190]  and try to work around
[24:32.190 --> 24:35.550]  some of the bugs in there.
[24:35.550 --> 24:36.550]  So we decided we needed
[24:36.550 --> 24:37.970]  an entire ground-up rewipe
[24:37.970 --> 24:39.910]  for Hashtag version two.
[24:41.630 --> 24:43.190]  And at some point,
[24:43.190 --> 24:43.970]  I got the idea
[24:44.950 --> 24:47.330]  to create a commercial fork of Hashcat
[24:47.330 --> 24:49.650]  with native distributed capabilities.
[24:49.850 --> 24:50.670]  And at the time,
[24:50.670 --> 24:52.690]  I was so fucking stoked on this idea.
[24:52.690 --> 24:54.610]  It seemed like the best idea
[24:55.170 --> 24:57.090]  in the world to me at the time.
[24:58.050 --> 25:01.330]  So we set out to do just that.
[25:02.150 --> 25:03.730]  Now this is right
[25:04.530 --> 25:05.210]  before
[25:05.890 --> 25:07.810]  Hashtag went open source,
[25:07.810 --> 25:09.930]  and then we started work on this
[25:09.930 --> 25:11.510]  literally the minute
[25:11.510 --> 25:13.890]  that Hashtag went open source.
[25:14.450 --> 25:16.210]  So we implemented in Golang
[25:16.210 --> 25:17.030]  in C
[25:18.010 --> 25:20.290]  with Postgres database.
[25:20.510 --> 25:21.730]  Now we chose Golang
[25:21.730 --> 25:23.290]  because of high concurrency
[25:23.950 --> 25:26.910]  with goroutines, right?
[25:27.090 --> 25:28.970]  So with Node.js,
[25:28.970 --> 25:31.150]  everything ran on a single CPU thread.
[25:31.150 --> 25:31.930]  And we wanted
[25:31.930 --> 25:34.490]  the opposite of that for version two, right?
[25:34.490 --> 25:35.990]  We're going to learn the lesson on that.
[25:35.990 --> 25:37.810]  So we wanted
[25:38.290 --> 25:39.770]  just lots and lots of goroutines
[25:39.770 --> 25:40.610]  to run everything
[25:40.610 --> 25:43.450]  so that we'd use all 24 threads
[25:43.450 --> 25:44.910]  plus on the box.
[25:46.870 --> 25:48.510]  And Golang actually also
[25:49.490 --> 25:51.130]  allows for pretty rapid development.
[25:51.130 --> 25:53.010]  So we thought it was a pretty good choice.
[25:54.850 --> 25:55.930]  But we still stuck
[25:55.930 --> 25:57.370]  with the traditional client server
[25:57.370 --> 25:58.690]  agent architecture.
[25:58.730 --> 26:01.570]  We have numerous clients
[26:01.570 --> 26:03.150]  and then one server
[26:03.150 --> 26:05.250]  and then however many agents.
[26:06.350 --> 26:07.570]  Meaning that that server
[26:07.570 --> 26:09.230]  is both a bottleneck
[26:09.230 --> 26:10.630]  and a single point of failure.
[26:11.910 --> 26:13.550]  So the first thing we had to do
[26:13.550 --> 26:15.430]  is we had to split Hashcat in two.
[26:15.990 --> 26:16.930]  We took everything
[26:16.930 --> 26:18.230]  that had to do with
[26:18.230 --> 26:21.290]  actually starting a cracking job
[26:21.290 --> 26:23.910]  and executing an OpenCL kernel
[26:23.910 --> 26:24.550]  on a GPU
[26:25.930 --> 26:26.150]  with the agent
[26:26.150 --> 26:27.610]  and nothing else.
[26:27.610 --> 26:29.830]  The agent was just barebones
[26:29.830 --> 26:31.030]  trimmed down Hashcat
[26:32.830 --> 26:34.210]  launch attack on GPU
[26:34.210 --> 26:35.990]  and that was it, nothing more.
[26:36.410 --> 26:37.490]  And everything else
[26:37.490 --> 26:39.750]  we placed into the server library.
[26:40.290 --> 26:42.350]  And then for any code
[26:42.350 --> 26:43.910]  that wasn't already in Hashcat
[26:43.910 --> 26:45.770]  such as the JSON API
[26:45.770 --> 26:47.510]  and the
[26:48.510 --> 26:49.110]  protobufs
[26:49.110 --> 26:50.630]  for agent server communication
[26:50.630 --> 26:51.810]  workload distribution
[26:55.930 --> 26:56.730]  fine-grained
[26:56.730 --> 26:59.390]  multi-role access control.
[26:59.470 --> 27:00.850]  All that was implemented in Go
[27:00.850 --> 27:03.650]  and then we just linked against the C libraries.
[27:03.650 --> 27:05.270]  And this effort actually
[27:05.270 --> 27:07.150]  laid the foundation for
[27:08.130 --> 27:10.410]  the Hashcat 4.0 refactor
[27:10.410 --> 27:12.150]  with libHashcat.
[27:12.610 --> 27:14.910]  And then because of how clunky
[27:14.910 --> 27:16.510]  our old web UI was
[27:16.510 --> 27:19.130]  we actually ditched it entirely
[27:19.130 --> 27:20.890]  for a cross-platform CLI
[27:20.890 --> 27:23.410]  which made our power users
[27:23.410 --> 27:24.590]  like really happy
[27:25.930 --> 27:28.290]  but our less experienced users
[27:28.290 --> 27:30.330]  were not very thrilled
[27:30.330 --> 27:31.390]  with that decision.
[27:31.390 --> 27:34.310]  They still wanted a GUI
[27:34.310 --> 27:36.830]  which, fair enough.
[27:38.090 --> 27:40.090]  So our plan initially
[27:40.090 --> 27:42.350]  was to maintain both Hashcat
[27:42.350 --> 27:44.590]  and Hashstack trees in parallel.
[27:44.830 --> 27:45.330]  And then
[27:46.570 --> 27:47.930]  if there was any features
[27:47.930 --> 27:49.230]  that we added to Hashstack
[27:49.230 --> 27:50.890]  or any new kernels that we wrote
[27:50.890 --> 27:52.850]  we would just backport those
[27:52.850 --> 27:55.510]  as we chose to Hashcat.
[27:58.550 --> 27:59.770]  This didn't really work out
[27:59.770 --> 28:00.790]  very well in practice though.
[28:00.790 --> 28:02.430]  So when we released version 2
[28:02.430 --> 28:04.750]  it was in sync with Hashcat 3.5
[28:05.230 --> 28:06.370]  but it didn't stay that way
[28:06.370 --> 28:07.370]  for very long.
[28:08.410 --> 28:09.510]  And of course we had to add
[28:09.610 --> 28:10.530]  a cross-platform GUI
[28:10.530 --> 28:11.290]  to the roadmap as well
[28:11.290 --> 28:12.790]  because we had significant
[28:12.790 --> 28:13.870]  feedback from our clients
[28:13.870 --> 28:15.390]  that they really wanted
[28:15.390 --> 28:16.290]  the goddamn GUI
[28:16.290 --> 28:16.990]  and they were not happy
[28:16.990 --> 28:18.370]  that we took away the web UI.
[28:19.890 --> 28:20.770]  It turns out
[28:20.770 --> 28:22.130]  this was the single dumbest
[28:22.130 --> 28:23.350]  motherfucking thing I've ever
[28:23.350 --> 28:24.450]  done in my life.
[28:25.590 --> 28:27.390]  I completely underestimated
[28:27.390 --> 28:28.450]  how much work was involved
[28:28.450 --> 28:30.410]  in creating a commercial fork
[28:30.410 --> 28:31.370]  of Hashcat.
[28:32.010 --> 28:33.230]  Like, genuinely
[28:36.470 --> 28:36.970]  I...
[28:37.770 --> 28:39.510]  Yeah, I really don't know
[28:39.510 --> 28:40.910]  how to convey how fucking stupid
[28:40.910 --> 28:42.310]  of an idea this was.
[28:43.110 --> 28:45.570]  We spent like 100%
[28:45.570 --> 28:46.650]  of our time just backporting
[28:47.210 --> 28:48.930]  things from Hashcat into Hashtag
[28:48.930 --> 28:50.110]  instead of actually adding new
[28:50.110 --> 28:51.650]  features or writing new kernels.
[28:52.290 --> 28:54.030]  Like, literally months of work
[28:54.550 --> 28:55.610]  you know, just backporting
[28:56.930 --> 28:58.130]  Hashcat commits.
[28:58.870 --> 29:00.690]  And it got really frustrating.
[29:00.910 --> 29:01.490]  You know, it's like,
[29:01.490 --> 29:02.650]  hey, are we working on these
[29:02.650 --> 29:03.690]  tickets? Are we working on these
[29:03.690 --> 29:04.930]  features? No, we're working on
[29:04.930 --> 29:05.770]  backports.
[29:05.990 --> 29:06.950]  You know, I got so sick of
[29:06.950 --> 29:07.930]  hearing working on backports,
[29:07.930 --> 29:09.330]  working on backports, like
[29:10.290 --> 29:11.230]  when are we ever going to be
[29:11.230 --> 29:12.230]  done with backports?
[29:13.430 --> 29:15.590]  And then when Adam refactored
[29:15.590 --> 29:17.950]  Hashcat for 4.0, that also
[29:17.950 --> 29:19.390]  meant that we had to refactor
[29:19.390 --> 29:20.990]  Hashtag as well.
[29:21.150 --> 29:22.550]  And we hadn't even finished
[29:22.550 --> 29:23.750]  implementing all the backports
[29:23.750 --> 29:25.410]  at the speed that Adam was, you
[29:25.410 --> 29:26.250]  know, committing things to
[29:26.250 --> 29:27.050]  Hashcat.
[29:27.170 --> 29:28.710]  So this was just like,
[29:28.710 --> 29:29.250]  absolutely
[29:30.510 --> 29:32.130]  unmaintainable, right?
[29:32.130 --> 29:34.250]  Like, fuck me.
[29:35.550 --> 29:36.870]  And then there was also this
[29:36.870 --> 29:38.010]  other little minor problem
[29:38.950 --> 29:40.610]  where we picked the wrong
[29:40.610 --> 29:41.570]  HTTP library
[29:42.230 --> 29:43.450]  for the API.
[29:44.130 --> 29:45.310]  And this basically
[29:45.310 --> 29:46.750]  limited the service capability
[29:46.750 --> 29:49.770]  to about 12 nodes.
[29:49.790 --> 29:51.290]  Now, 12 nodes
[29:51.290 --> 29:52.830]  might seem like a lot.
[29:52.830 --> 29:54.510]  That's 96 GPUs.
[29:54.510 --> 29:55.270]  You know, that's a pretty
[29:55.270 --> 29:57.670]  respectable size cluster.
[29:58.890 --> 30:01.310]  The problem is this
[30:01.310 --> 30:03.050]  particular client had
[30:03.050 --> 30:05.950]  ordered 320 GPUs,
[30:05.950 --> 30:07.230]  40 nodes.
[30:07.510 --> 30:09.190]  And you can't really
[30:09.190 --> 30:11.250]  deliver a 40 node client
[30:11.250 --> 30:14.290]  to a cluster with a
[30:14.290 --> 30:15.970]  server that can only
[30:15.970 --> 30:17.230]  really handle
[30:17.230 --> 30:22.360]  12 nodes.
[30:22.360 --> 30:23.300]  So this required
[30:24.680 --> 30:26.280]  not a complete rewrite,
[30:26.440 --> 30:27.840]  but a damn near complete
[30:27.840 --> 30:28.320]  rewrite time.
[30:28.320 --> 30:29.500]  Fuck, because we use that
[30:29.500 --> 30:30.300]  HTTP library
[30:31.780 --> 30:32.620]  extensively throughout the
[30:32.620 --> 30:34.400]  code.
[30:34.400 --> 30:35.440]  And there was no, like, drop
[30:35.440 --> 30:36.140]  in replacement.
[30:36.140 --> 30:37.200]  So basically anywhere where
[30:37.200 --> 30:38.800]  we were dealing with
[30:38.800 --> 30:39.880]  HTTP,
[30:39.880 --> 30:41.220]  we had to rip out the old
[30:41.220 --> 30:42.800]  library and then,
[30:42.800 --> 30:43.200]  you know,
[30:44.060 --> 30:44.960]  add in all the stuff for
[30:44.960 --> 30:45.840]  the new library.
[30:45.960 --> 30:47.000]  But they did resolve the
[30:47.000 --> 30:47.540]  issue.
[30:48.320 --> 30:49.500]  It was just really painful
[30:49.500 --> 30:50.540]  at the time to have to
[30:50.540 --> 30:51.400]  mess with that.
[30:53.420 --> 30:54.600]  But the server
[30:54.600 --> 30:56.740]  still continued to be a
[30:56.740 --> 30:57.780]  single point of failure
[30:57.780 --> 30:58.820]  and still proved to be a
[30:58.820 --> 31:00.180]  source of bottlenecks.
[31:00.180 --> 31:01.540]  As we, you know,
[31:01.540 --> 31:02.150]  tried to build
[31:03.220 --> 31:04.360]  larger and larger
[31:05.160 --> 31:06.000]  clusters,
[31:06.580 --> 31:07.000]  we
[31:07.000 --> 31:07.640]  kept running into
[31:07.640 --> 31:09.160]  problems with the
[31:09.640 --> 31:10.060]  server
[31:10.200 --> 31:10.660]  being a bottleneck
[31:10.760 --> 31:11.380]  and a single point of
[31:11.380 --> 31:12.340]  failure.
[31:12.340 --> 31:13.080]  And we also had clients
[31:13.080 --> 31:14.200]  coming to us asking us
[31:14.640 --> 31:15.520]  if they could order
[31:15.520 --> 31:16.400]  multiple cluster
[31:16.400 --> 31:17.780]  controllers and cluster
[31:17.780 --> 31:18.800]  them together,
[31:18.800 --> 31:19.240]  you know, either
[31:19.240 --> 31:19.960]  high availability
[31:19.960 --> 31:21.860]  or, like,
[31:21.860 --> 31:22.700]  load balance them
[31:22.700 --> 31:23.540]  or something.
[31:24.560 --> 31:24.980]  And
[31:26.680 --> 31:27.660]  we didn't have a
[31:27.660 --> 31:28.380]  solution for that,
[31:28.380 --> 31:29.140]  but I started thinking
[31:29.280 --> 31:32.300]  about a solution for that.
[31:32.300 --> 31:33.220]  And again, like,
[31:33.220 --> 31:33.980]  40% of our clients
[31:33.980 --> 31:34.660]  were just completely
[31:34.660 --> 31:35.620]  pissed that we had no
[31:35.620 --> 31:40.280]  GUI.
[31:40.280 --> 31:40.480]  So
[31:40.480 --> 31:41.340]  it got to the point
[31:41.340 --> 31:42.020]  where it's like,
[31:41.080 --> 31:42.080]  let's just
[31:42.540 --> 31:44.080]  rethink, like,
[31:44.080 --> 31:45.000]  the entire fucking thing
[31:45.000 --> 31:46.140]  and just, like,
[31:46.140 --> 31:47.000]  go as far away from
[31:47.000 --> 31:48.080]  these mistakes as we
[31:48.080 --> 31:49.160]  could possibly get.
[31:49.220 --> 31:50.080]  Like, we never, ever
[31:50.080 --> 31:50.780]  want to make any of
[31:50.780 --> 31:51.860]  these mistakes ever again.
[31:51.860 --> 31:52.420]  We've learned a lot of
[31:52.420 --> 31:53.960]  lessons from, you know,
[31:53.960 --> 31:54.720]  what we've done over
[31:54.720 --> 31:55.920]  the last seven years.
[31:57.340 --> 31:58.640]  So, but we need
[31:58.640 --> 31:59.360]  more than just a
[31:59.360 --> 32:00.120]  complete ground-up rewrite.
[32:00.120 --> 32:01.220]  Like, we need an entire
[32:01.220 --> 32:01.640]  paradigm shift.
[32:01.640 --> 32:03.100]  Like, we need to
[32:03.100 --> 32:04.020]  rethink this entire
[32:05.480 --> 32:05.880]  problem
[32:06.640 --> 32:07.040]  entirely
[32:07.600 --> 32:08.000]  and
[32:08.000 --> 32:08.620]  come up with
[32:08.620 --> 32:09.300]  something that's
[32:09.300 --> 32:09.980]  completely different
[32:09.980 --> 32:10.460]  from what's ever been
[32:10.460 --> 32:11.120]  done.
[32:11.940 --> 32:12.820]  So we need to
[32:12.820 --> 32:13.740]  eliminate the single
[32:13.740 --> 32:14.400]  point of failure of
[32:14.400 --> 32:15.180]  the server.
[32:15.300 --> 32:16.640]  We need to enable,
[32:16.640 --> 32:17.600]  like, actual infinite
[32:17.600 --> 32:18.440]  horizontal scaling
[32:18.440 --> 32:19.440]  because, again,
[32:19.440 --> 32:20.480]  we're, you know,
[32:20.480 --> 32:21.740]  we always try to
[32:21.740 --> 32:23.440]  strive for, like,
[32:23.440 --> 32:24.460]  warehouse as a
[32:24.460 --> 32:25.360]  computer, right?
[32:25.360 --> 32:26.900]  We're looking at
[32:26.900 --> 32:27.560]  warehouse scale
[32:27.560 --> 32:28.300]  computing as, like,
[32:28.300 --> 32:28.880]  the target that we're
[32:28.880 --> 32:31.360]  trying to hit.
[32:31.360 --> 32:32.660]  There should never be
[32:32.660 --> 32:34.860]  any, you know,
[32:34.860 --> 32:35.560]  limiting factors
[32:35.560 --> 32:36.540]  outside of budget
[32:36.540 --> 32:38.280]  for, like, how many
[32:38.280 --> 32:38.860]  nodes we can
[32:38.860 --> 32:39.280]  support in
[32:39.280 --> 32:39.560]  HashtagWrite.
[32:40.220 --> 32:41.160]  And then, of course,
[32:41.160 --> 32:42.040]  the other big requirement
[32:42.040 --> 32:42.700]  is we have to actually
[32:42.700 --> 32:43.560]  move at the pace of
[32:43.560 --> 32:44.740]  Hashtag development.
[32:46.040 --> 32:47.620]  We have to,
[32:47.620 --> 32:48.380]  you know, as Adam
[32:48.380 --> 32:49.500]  commits things to
[32:49.500 --> 32:50.420]  Hashtag, we need to
[32:50.420 --> 32:51.040]  have those
[32:51.960 --> 32:53.280]  not instantly available
[32:53.280 --> 32:54.440]  in Hashtag, but within
[32:54.600 --> 32:55.280]  a very reasonable
[32:55.280 --> 32:56.260]  time frame.
[32:58.880 --> 32:59.320]  So
[33:00.000 --> 33:01.000]  the very first
[33:01.000 --> 33:01.760]  idea that we came up
[33:01.760 --> 33:02.620]  with was let's not
[33:02.620 --> 33:03.540]  do a traditional
[33:04.540 --> 33:06.000]  client-server-agent
[33:06.740 --> 33:07.460]  model.
[33:08.440 --> 33:09.540]  And let's not even
[33:09.540 --> 33:10.340]  try to cluster the
[33:10.340 --> 33:11.400]  cluster controllers.
[33:11.740 --> 33:12.780]  Let's just eliminate
[33:12.780 --> 33:13.620]  the cluster controllers
[33:13.620 --> 33:14.800]  entirely and make
[33:14.800 --> 33:15.580]  everything peer-to-peer
[33:15.580 --> 33:16.740]  where all nodes
[33:16.740 --> 33:17.160]  are equal
[33:17.860 --> 33:18.620]  and there are no
[33:18.620 --> 33:19.860]  dedicated servers.
[33:20.860 --> 33:21.100]  And
[33:21.920 --> 33:22.960]  the biggest requirement
[33:22.960 --> 33:23.860]  to this is that
[33:23.860 --> 33:24.660]  work can be submitted
[33:24.660 --> 33:25.760]  to any node.
[33:26.460 --> 33:27.540]  So let's say you have,
[33:27.540 --> 33:28.620]  like, you know, 20 users
[33:28.620 --> 33:29.780]  and you have,
[33:29.780 --> 33:31.120]  you know,
[33:31.120 --> 33:32.000]  10 boxes.
[33:32.000 --> 33:32.780]  Like, there shouldn't
[33:32.780 --> 33:33.720]  be just one node that
[33:33.720 --> 33:34.900]  all of them have to hit.
[33:34.900 --> 33:35.700]  They should be able
[33:35.700 --> 33:36.420]  to hit any node
[33:36.420 --> 33:37.160]  in the cluster
[33:37.160 --> 33:38.400]  and, you know, have
[33:38.400 --> 33:39.160]  the same view
[33:39.160 --> 33:40.600]  from any node.
[33:41.880 --> 33:42.780]  And it also needs to
[33:42.780 --> 33:43.400]  be more user-friendly
[33:43.400 --> 33:44.140]  than ever.
[33:44.400 --> 33:45.400]  And also needs to
[33:45.400 --> 33:46.340]  run on Windows.
[33:48.100 --> 33:48.960]  I'll tell you why
[33:48.960 --> 33:49.640]  in a minute.
[33:52.240 --> 33:53.380]  So set out to,
[33:53.380 --> 33:54.120]  like, figure out
[33:54.120 --> 33:54.660]  how we're going to
[33:54.660 --> 33:55.320]  implement this.
[33:55.320 --> 33:55.820]  Because we have the
[33:55.820 --> 33:57.280]  high-level requirements,
[33:57.280 --> 33:58.040]  right? Like, we know
[33:58.040 --> 33:58.760]  it needs to be
[33:58.760 --> 33:59.380]  peer-to-peer.
[33:59.380 --> 34:00.140]  We know all the nodes
[34:00.140 --> 34:01.060]  need to be equal.
[34:01.240 --> 34:02.520]  You know, all that shit.
[34:03.040 --> 34:03.800]  But how are we actually
[34:03.800 --> 34:04.740]  going to do this?
[34:05.600 --> 34:05.960]  So
[34:06.840 --> 34:07.560]  pretty obviously
[34:07.560 --> 34:08.060]  we're going to need
[34:08.060 --> 34:08.540]  some sort of
[34:08.540 --> 34:09.880]  distributed state machine.
[34:10.040 --> 34:10.680]  Right?
[34:11.460 --> 34:12.400]  And that's just
[34:12.400 --> 34:13.280]  straightforward
[34:13.280 --> 34:14.520]  and self-explanatory.
[34:14.780 --> 34:16.360]  But, you know,
[34:16.360 --> 34:17.180]  like, we just
[34:17.180 --> 34:17.600]  throw a message
[34:17.600 --> 34:18.040]  queue at it
[34:18.040 --> 34:18.520]  and some PubSub
[34:18.520 --> 34:20.400]  and, you know,
[34:20.400 --> 34:21.500]  Bob's your uncle.
[34:22.160 --> 34:22.860]  But what about
[34:22.860 --> 34:23.860]  disagreements?
[34:24.060 --> 34:25.460]  Or, you know,
[34:25.460 --> 34:26.060]  maybe even more
[34:26.060 --> 34:26.760]  than disagreements,
[34:26.760 --> 34:27.360]  what about, like,
[34:27.360 --> 34:28.680]  an actual, you know,
[34:28.680 --> 34:29.060]  rogue
[34:29.900 --> 34:31.100]  cluster member?
[34:31.220 --> 34:33.100]  Or, you know,
[34:33.980 --> 34:34.380]  a broken
[34:34.820 --> 34:35.520]  cluster node
[34:35.520 --> 34:36.560]  that's maybe not
[34:36.560 --> 34:36.980]  necessarily
[34:36.980 --> 34:37.340]  intentionally
[34:37.340 --> 34:38.180]  malicious,
[34:38.180 --> 34:39.000]  but it certainly,
[34:39.000 --> 34:39.300]  you know,
[34:39.300 --> 34:41.180]  has, you know,
[34:41.180 --> 34:41.820]  the appearance of
[34:41.820 --> 34:42.700]  malicious behavior.
[34:42.700 --> 34:43.500]  Right?
[34:44.620 --> 34:45.600]  So I started
[34:45.600 --> 34:46.320]  coming up with
[34:46.940 --> 34:47.560]  solutions for
[34:47.560 --> 34:48.060]  this.
[34:48.680 --> 34:49.280]  And
[34:49.980 --> 34:50.960]  I somehow
[34:50.960 --> 34:51.740]  got into my
[34:51.740 --> 34:53.220]  head, like,
[34:53.220 --> 34:54.200]  dude,
[34:54.200 --> 34:54.860]  what if
[34:54.860 --> 34:55.600]  computers could
[34:55.600 --> 34:56.580]  vote?
[34:57.000 --> 34:57.840]  Right?
[34:57.840 --> 34:58.980]  Like,
[35:01.520 --> 35:02.360]  you just
[35:02.360 --> 35:02.840]  make it to
[35:02.840 --> 35:03.300]  where, like,
[35:03.300 --> 35:03.580]  you know,
[35:03.580 --> 35:04.420]  if there's a
[35:04.420 --> 35:05.260]  dispute,
[35:06.100 --> 35:06.440]  then one of
[35:06.440 --> 35:06.920]  the nodes,
[35:06.920 --> 35:07.520]  like,
[35:07.520 --> 35:08.140]  requests a vote,
[35:08.140 --> 35:08.740]  right?
[35:09.060 --> 35:10.160]  And so they take a vote
[35:10.160 --> 35:11.060]  and majority wins.
[35:11.060 --> 35:11.820]  And if there's a tie,
[35:11.820 --> 35:12.360]  like we have an even
[35:12.360 --> 35:13.200]  number of nodes in the
[35:13.200 --> 35:14.600]  cluster, then,
[35:14.600 --> 35:14.700]  like,
[35:14.700 --> 35:15.420]  literally fucking
[35:15.420 --> 35:16.640]  flip a coin,
[35:16.640 --> 35:17.140]  you know,
[35:17.140 --> 35:17.640]  randomly,
[35:17.640 --> 35:18.500]  or play rock,
[35:18.500 --> 35:18.720]  paper,
[35:18.720 --> 35:19.220]  scissors.
[35:19.360 --> 35:19.980]  Like,
[35:19.980 --> 35:20.780]  why not?
[35:22.100 --> 35:22.960]  And if a peer
[35:22.960 --> 35:23.660]  doesn't accept the
[35:23.660 --> 35:24.440]  results, like,
[35:24.440 --> 35:24.880]  then the rest of
[35:24.880 --> 35:25.400]  the cluster just
[35:25.400 --> 35:26.380]  shuns that peer.
[35:26.800 --> 35:27.400]  And I started
[35:27.400 --> 35:27.820]  thinking about,
[35:27.820 --> 35:27.940]  like,
[35:27.940 --> 35:28.920]  is this actually,
[35:28.920 --> 35:29.040]  like,
[35:29.100 --> 35:29.380]  a thing?
[35:29.380 --> 35:29.500]  Like,
[35:29.500 --> 35:30.080]  can we actually do
[35:30.080 --> 35:30.380]  this?
[35:30.380 --> 35:30.860]  Am I,
[35:30.860 --> 35:31.160]  like,
[35:31.160 --> 35:31.760]  completely fucking
[35:31.760 --> 35:32.400]  stoned?
[35:34.600 --> 35:35.420]  It turns out that
[35:35.420 --> 35:35.780]  I really didn't have
[35:35.780 --> 35:36.160]  to put that much
[35:36.160 --> 35:36.640]  more thought into
[35:36.640 --> 35:37.340]  it because this
[35:37.340 --> 35:38.120]  already exists.
[35:38.120 --> 35:38.320]  Like,
[35:38.320 --> 35:39.180]  what I thought up
[35:39.660 --> 35:40.820]  is actually a
[35:40.820 --> 35:41.700]  thing called the
[35:41.700 --> 35:42.480]  wrapped consensus
[35:42.480 --> 35:43.940]  protocol.
[35:43.940 --> 35:45.040]  Now,
[35:45.040 --> 35:45.920]  it's cool that it
[35:45.920 --> 35:46.420]  already exists because
[35:46.420 --> 35:46.880]  that,
[35:46.880 --> 35:47.220]  you know,
[35:47.220 --> 35:47.460]  means we don't
[35:47.460 --> 35:47.700]  have to do a
[35:47.700 --> 35:48.080]  whole lot to
[35:48.080 --> 35:48.520]  implement it.
[35:48.520 --> 35:49.140]  But it's also cool
[35:49.140 --> 35:49.980]  because it kind of
[35:50.480 --> 35:51.200]  validated,
[35:51.200 --> 35:51.940]  you know,
[35:51.940 --> 35:52.860]  my ideas and theories
[35:52.860 --> 35:53.560]  on how to make this
[35:53.560 --> 35:54.260]  work.
[35:55.080 --> 35:55.980]  And told me that I
[35:55.980 --> 35:56.260]  wasn't just
[35:56.260 --> 35:56.780]  completely fucking
[35:56.780 --> 35:57.400]  insane.
[35:58.380 --> 35:59.600]  And we found
[35:59.600 --> 36:00.800]  several implementations
[36:00.800 --> 36:01.420]  of the wrapped
[36:01.420 --> 36:02.460]  consensus protocol.
[36:03.640 --> 36:04.860]  And the one that
[36:04.860 --> 36:05.900]  we ended up liking
[36:05.900 --> 36:06.540]  the most and
[36:06.540 --> 36:07.500]  settled on is
[36:07.500 --> 36:08.100]  Akka,
[36:08.100 --> 36:08.960]  which runs on
[36:08.960 --> 36:09.620]  the JVM.
[36:09.620 --> 36:09.980]  It's written in
[36:09.980 --> 36:10.660]  Scala.
[36:11.140 --> 36:11.920]  Now,
[36:12.580 --> 36:14.180]  I hate Java.
[36:14.180 --> 36:15.020]  I have shunned
[36:15.020 --> 36:16.980]  Java for damn
[36:16.980 --> 36:17.640]  near my entire
[36:17.640 --> 36:18.500]  life.
[36:18.740 --> 36:19.100]  You know,
[36:19.100 --> 36:20.300]  I completely
[36:20.300 --> 36:21.040]  condemn
[36:22.280 --> 36:23.700]  Oracle JVM.
[36:26.660 --> 36:27.700]  But,
[36:27.700 --> 36:28.860]  there is
[36:28.860 --> 36:29.900]  OpenJDK.
[36:29.900 --> 36:30.420]  We don't have
[36:30.420 --> 36:31.360]  to use Oracle.
[36:31.580 --> 36:33.020]  And we don't
[36:33.020 --> 36:33.400]  have to write
[36:33.400 --> 36:34.160]  in Java.
[36:34.160 --> 36:34.620]  We can use
[36:34.620 --> 36:35.060]  Scala and
[36:35.060 --> 36:35.860]  Kotlin.
[36:36.780 --> 36:37.420]  And since
[36:37.420 --> 36:37.840]  I've started
[36:37.840 --> 36:38.200]  to work on
[36:38.200 --> 36:38.480]  this,
[36:38.480 --> 36:39.200]  I actually
[36:39.200 --> 36:40.020]  really like
[36:40.020 --> 36:40.920]  Kotlin.
[36:41.780 --> 36:42.600]  I don't
[36:42.600 --> 36:43.500]  really have too
[36:43.500 --> 36:43.940]  many negative
[36:43.940 --> 36:44.480]  things to say
[36:44.480 --> 36:44.860]  about it.
[36:44.860 --> 36:45.160]  It's actually
[36:45.160 --> 36:46.040]  kind of fun
[36:46.040 --> 36:47.180]  to work with.
[36:48.420 --> 36:48.720]  And then
[36:48.720 --> 36:49.620]  Akka also
[36:49.620 --> 36:50.420]  has some
[36:51.400 --> 36:52.220]  plugins or
[36:52.220 --> 36:54.000]  modules that,
[36:54.780 --> 36:55.760]  you know,
[36:55.760 --> 36:56.600]  help us even
[36:56.600 --> 36:57.160]  further.
[36:57.160 --> 36:57.620]  Such as
[36:57.620 --> 36:57.880]  Akka
[36:57.880 --> 36:58.880]  persistence.
[36:58.880 --> 36:59.260]  What Akka
[36:59.260 --> 37:00.040]  persistence does,
[37:00.040 --> 37:00.440]  it takes our
[37:00.440 --> 37:01.080]  distributed state
[37:01.080 --> 37:02.100]  machine,
[37:02.100 --> 37:03.000]  which is
[37:03.000 --> 37:03.200]  normally
[37:03.200 --> 37:03.700]  resident in
[37:03.700 --> 37:04.020]  memory,
[37:04.020 --> 37:04.480]  and persists
[37:04.480 --> 37:05.320]  it to
[37:05.320 --> 37:06.120]  disk.
[37:06.600 --> 37:07.040]  And then
[37:07.040 --> 37:08.020]  Akka
[37:08.020 --> 37:08.380]  distributed
[37:08.380 --> 37:08.860]  data, of
[37:08.860 --> 37:09.180]  course, would
[37:09.180 --> 37:09.660]  be like what
[37:09.660 --> 37:10.500]  implements the
[37:10.500 --> 37:11.220]  distributed state
[37:11.220 --> 37:11.960]  machine.
[37:12.420 --> 37:13.400]  And then on
[37:13.400 --> 37:13.880]  top of all
[37:13.880 --> 37:14.840]  of that,
[37:14.840 --> 37:16.120]  we built a
[37:16.120 --> 37:16.700]  custom
[37:16.700 --> 37:17.900]  multicast-based
[37:17.900 --> 37:19.200]  protocol to
[37:19.200 --> 37:20.040]  enable cluster
[37:20.040 --> 37:21.000]  nodes to
[37:21.000 --> 37:21.240]  automatically
[37:21.240 --> 37:21.540]  discover each
[37:21.540 --> 37:21.940]  other on
[37:21.940 --> 37:22.700]  the network
[37:22.700 --> 37:23.360]  and then
[37:23.360 --> 37:23.480]  automatically
[37:23.480 --> 37:24.220]  join the
[37:25.020 --> 37:25.560]  cluster.
[37:25.560 --> 37:25.980]  Now, we
[37:25.980 --> 37:26.820]  made this opt-in
[37:26.820 --> 37:27.080]  because obviously
[37:27.080 --> 37:27.620]  that's kind of
[37:36.020 --> 37:37.020]  a security
[37:27.620 --> 37:28.120]  risk if
[37:28.120 --> 37:28.800]  anybody can
[37:28.800 --> 37:29.420]  just join the
[37:29.420 --> 37:30.640]  cluster at
[37:30.640 --> 37:31.400]  will.
[37:33.880 --> 37:34.920]  But for
[37:34.920 --> 37:35.780]  some scenarios
[37:35.780 --> 37:36.320]  like where
[37:36.320 --> 37:36.820]  all of your
[37:36.820 --> 37:37.300]  nodes are on
[37:37.400 --> 37:37.800]  a dedicated
[37:38.660 --> 37:39.380]  subnet that's
[37:39.380 --> 37:40.140]  properly firewalled
[37:40.140 --> 37:40.600]  off and you
[37:40.600 --> 37:41.120]  have proper
[37:41.120 --> 37:41.780]  layer 3 and
[37:41.780 --> 37:42.420]  layer 4 access
[37:42.420 --> 37:44.440]  controls,
[37:45.760 --> 37:47.060]  this would be a
[37:47.060 --> 37:47.420]  solution that would
[37:47.420 --> 37:48.540]  be really handy.
[37:48.540 --> 37:49.720]  So, again,
[37:49.720 --> 37:50.700]  thinking about
[37:50.700 --> 37:51.440]  warehouse-scale
[37:51.440 --> 37:52.620]  computing or
[37:53.200 --> 37:54.080]  data center as
[37:54.080 --> 37:55.280]  a computer,
[37:55.280 --> 37:55.840]  let's say you
[37:55.840 --> 37:56.160]  just have racks
[37:56.160 --> 37:56.720]  and racks and
[37:56.720 --> 37:57.460]  racks of compute
[37:57.460 --> 37:58.180]  nodes, you would
[37:58.180 --> 38:00.120]  just unbox one,
[38:00.120 --> 38:00.740]  literally throw it
[38:00.740 --> 38:01.180]  in the rack and
[38:01.180 --> 38:02.560]  power it on, and
[38:02.560 --> 38:03.220]  it would discover
[38:03.220 --> 38:04.740]  the cluster as
[38:04.740 --> 38:05.040]  soon as it's
[38:05.040 --> 38:05.680]  powered on and
[38:05.680 --> 38:06.220]  join the cluster
[38:06.220 --> 38:06.840]  and start doing
[38:06.840 --> 38:07.500]  work.
[38:07.660 --> 38:08.380]  That's kind of
[38:08.380 --> 38:08.860]  the vision that we
[38:08.860 --> 38:09.840]  have for this.
[38:11.420 --> 38:12.060]  And, of course,
[38:12.060 --> 38:12.420]  since it's all
[38:12.420 --> 38:13.580]  peer-to-peer,
[38:13.580 --> 38:15.120]  that enables it
[38:15.120 --> 38:15.900]  to actually be
[38:15.900 --> 38:16.400]  just literally
[38:16.400 --> 38:17.280]  that simple.
[38:19.720 --> 38:20.580]  But, of course,
[38:20.580 --> 38:21.760]  the default is
[38:21.760 --> 38:22.260]  you manually
[38:22.260 --> 38:22.980]  specify which
[38:22.980 --> 38:23.800]  peers you want
[38:23.800 --> 38:24.220]  to be part of
[38:24.220 --> 38:25.040]  the swarm.
[38:29.110 --> 38:29.410]  Okay.
[38:29.410 --> 38:31.070]  And then, in
[38:31.070 --> 38:32.050]  April this year,
[38:32.050 --> 38:32.470]  we had the
[38:32.470 --> 38:33.110]  unique opportunity
[38:33.110 --> 38:33.410]  to present
[38:33.410 --> 38:34.170]  itself to
[38:34.170 --> 38:34.710]  purchase
[38:34.710 --> 38:35.950]  loft crack.
[38:37.210 --> 38:38.090]  Dildog approached
[38:38.090 --> 38:38.610]  me and we
[38:38.610 --> 38:39.430]  entered negotiations
[38:39.430 --> 38:41.610]  and we
[38:43.050 --> 38:43.630]  purchased it
[38:43.630 --> 38:43.990]  earlier this
[38:43.990 --> 38:44.610]  year primarily
[38:46.290 --> 38:47.370]  for the
[38:47.370 --> 38:49.310]  GUI, which
[38:49.310 --> 38:50.330]  is written
[38:50.330 --> 38:50.890]  in C++
[38:50.890 --> 38:52.710]  using QT
[38:52.710 --> 38:53.350]  as the
[38:53.350 --> 38:53.690]  windowing
[38:53.690 --> 38:54.570]  framework.
[38:54.810 --> 38:55.170]  And while
[38:55.170 --> 38:55.510]  it's not
[38:55.510 --> 38:55.830]  currently
[38:55.830 --> 38:56.670]  cross-platform,
[38:56.670 --> 38:57.070]  that's a
[38:57.070 --> 38:57.870]  really easy
[38:57.870 --> 38:59.250]  platform, or
[38:59.370 --> 39:00.470]  that's a really
[39:00.470 --> 39:00.910]  easy language
[39:00.910 --> 39:01.310]  and library
[39:01.310 --> 39:01.910]  to make
[39:01.910 --> 39:08.130]  cross-platform.
[39:08.130 --> 39:09.310]  So, instead
[39:09.310 --> 39:09.810]  of
[39:09.810 --> 39:10.070]  hashtag
[39:10.070 --> 39:11.230]  version 3,
[39:11.230 --> 39:11.750]  we now
[39:11.750 --> 39:12.590]  have loft
[39:13.970 --> 39:14.410]  crack version
[39:14.410 --> 39:14.930]  8.
[39:14.930 --> 39:16.710]  And while
[39:16.710 --> 39:17.110]  loft crack
[39:17.110 --> 39:17.250]  7 uses
[39:17.250 --> 39:17.810]  John the
[39:17.810 --> 39:18.810]  Ripper as
[39:18.810 --> 39:19.430]  a backend,
[39:19.430 --> 39:20.470]  we've ripped
[39:20.470 --> 39:20.690]  all that out
[39:20.690 --> 39:21.630]  and we are
[39:21.630 --> 39:22.790]  in the process
[39:22.830 --> 39:23.830]  of integrating
[39:23.830 --> 39:24.830]  what was
[39:22.790 --> 39:23.390]  hashtag version
[39:23.390 --> 39:24.770]  3 as the
[39:24.770 --> 39:25.450]  new backend
[39:25.450 --> 39:26.130]  for loft crack
[39:26.130 --> 39:26.710]  8.
[39:28.670 --> 39:29.710]  And the
[39:29.710 --> 39:31.010]  final thing
[39:31.010 --> 39:31.330]  we're doing
[39:31.330 --> 39:32.170]  with this is
[39:32.170 --> 39:32.870]  we are dropping
[39:32.870 --> 39:33.590]  the
[39:33.590 --> 39:34.070]  TeraHash
[39:34.070 --> 39:34.330]  hardware
[39:34.330 --> 39:35.310]  requirement.
[39:35.490 --> 39:36.710]  Traditionally,
[39:36.710 --> 39:37.550]  you know, we
[39:37.550 --> 39:39.650]  do not let
[39:39.650 --> 39:40.810]  people purchase
[39:40.810 --> 39:42.270]  hashtag as a
[39:42.270 --> 39:42.950]  standalone product.
[39:42.950 --> 39:43.550]  We say, like,
[39:43.550 --> 39:44.070]  you know, the only
[39:44.070 --> 39:44.750]  way to get
[39:44.750 --> 39:45.650]  hashtag is to
[39:46.010 --> 39:46.410]  buy a
[39:46.410 --> 39:48.670]  TeraHash
[39:48.670 --> 39:49.310]  appliance and
[39:49.790 --> 39:50.790]  it comes
[39:50.790 --> 39:51.790]  preinstalled
[39:49.310 --> 39:49.690]  and that's the
[39:49.690 --> 39:50.050]  only way to
[39:50.050 --> 39:50.870]  get it.
[39:51.430 --> 39:52.230]  But loft crack
[39:52.230 --> 39:52.810]  has always been
[39:52.810 --> 39:53.890]  sold standalone.
[39:54.350 --> 39:56.030]  So we are changing
[39:56.030 --> 39:56.730]  the model now
[39:57.310 --> 39:58.250]  to where,
[39:59.150 --> 39:59.730]  you know,
[39:59.730 --> 40:00.170]  keeping with
[40:00.170 --> 40:00.570]  loft crack
[40:00.570 --> 40:01.030]  being sold
[40:01.030 --> 40:01.490]  standalone,
[40:01.490 --> 40:01.710]  we're going to
[40:01.710 --> 40:02.290]  be selling
[40:03.710 --> 40:04.330]  loft crack
[40:04.330 --> 40:05.070]  8 as a
[40:05.070 --> 40:05.970]  standalone product
[40:05.970 --> 40:06.590]  without the
[40:06.590 --> 40:10.470]  TeraHash
[40:10.470 --> 40:10.530]  hardware
[40:10.530 --> 40:10.770]  requirement.
[40:10.770 --> 40:12.010]  So, yeah,
[40:12.010 --> 40:13.150]  that's where
[40:13.630 --> 40:14.470]  we are now
[40:14.470 --> 40:15.150]  with development.
[40:15.150 --> 40:15.290]  That's how the
[40:17.310 --> 40:18.310]  product has
[40:15.290 --> 40:15.750]  evolved over
[40:15.750 --> 40:16.270]  time.
[40:16.270 --> 40:16.870]  And that's some
[40:16.870 --> 40:17.410]  of the
[40:17.410 --> 40:18.750]  challenges that
[40:18.750 --> 40:19.490]  we have faced
[40:19.490 --> 40:21.670]  trying to
[40:22.830 --> 40:23.690]  obtain our
[40:23.690 --> 40:24.390]  goal of
[40:24.390 --> 40:24.890]  true
[40:24.890 --> 40:25.750]  warehouse scale
[40:25.750 --> 40:26.730]  computing.
[40:26.830 --> 40:27.730]  Thank you and
[40:27.730 --> 40:28.270]  hope you've enjoyed
[40:28.270 --> 40:29.010]  this.
